Competence By Design: a transformational national model of time-variable competency-based postgraduate medical education.

Postgraduate medical education is an essential societal enterprise that prepares highly skilled physicians for the health workforce. In recent years, PGME systems have been criticized worldwide for problems with variable graduate abilities, concerns about patient safety, and issues with teaching and assessment methods. In response, competency based medical education approaches, with an emphasis on graduate outcomes, have been proposed as the direction for 21st century health profession education. However, there are few published models of large-scale implementation of these approaches. We describe the rationale and design for a national, time-variable competency-based multi-specialty system for postgraduate medical education called Competence by Design. Fourteen innovations were bundled to create this new system, using the Van Melle Core Components of competency based medical education as the basis for the transformation. The successful execution of this transformational training system shows competency based medical education can be implemented at scale. The lessons learned in the early implementation of Competence by Design can inform competency based medical education innovation efforts across professions worldwide.

Enabling Implementation of Competency Based Medical Education through an Outcomes-Focused Accreditation System.

Competency based medical education is being adopted around the world. Accreditation plays a vital role as an enabler in the adoption and implementation of competency based medical education, but little has been published about how the design of an accreditation system facilitates this transformation. The Canadian postgraduate medical education environment has recently transitioned to an outcomes-based accreditation system in parallel with the adoption of competency based medical education. Using the Canadian example, we characterize four features of an accreditation system that can facilitate the implementation of competency based medical education: theoretical underpinning, quality focus, accreditation standards, and accreditation processes. Alignment of the underlying educational theories within the accreditation system and educational paradigm drives change in a consistent and desired direction. An accreditation system that prioritizes quality improvement over quality assurance promotes educational system development and progressive change. Accreditation standards that achieve the difficult balance of being sufficiently detailed yet flexible foster a high fidelity of implementation without stifling innovation. Finally, accreditation processes that recognize the change process, encourage program development, and are not overly punitive all enable the implementation of competency based medical education. We also discuss the ways in which accreditation can simultaneously hinder the implementation of this approach. As education bodies adopt competency based medical education, particular attention should be paid to the role that accreditation plays in successful implementation.

Using Natural Language Processing to Evaluate the Quality of Supervisor Narrative Comments in Competency-Based Medical Education.

PURPOSE: Learner development and promotion rely heavily on narrative assessment comments, but narrative assessment quality is rarely evaluated in medical education. Educators have developed tools such as the Quality of Assessment for Learning (QuAL) tool to evaluate the quality of narrative assessment comments; however, scoring the comments generated in medical education assessment programs is time intensive. The authors developed a natural language processing (NLP) model for applying the QuAL score to narrative supervisor comments. METHOD: Samples of 2,500 Entrustable Professional Activities assessments were randomly extracted and deidentified from the McMaster (1,250 comments) and Saskatchewan (1,250 comments) emergency medicine (EM) residency training programs during the 2019-2020 academic year. Comments were rated using the QuAL score by 25 EM faculty members and 25 EM residents. The results were used to develop and test an NLP model to predict the overall QuAL score and QuAL subscores. RESULTS: All 50 raters completed the rating exercise. Approximately 50% of the comments had perfect agreement on the QuAL score, with the remaining resolved by the study authors. Creating a meaningful suggestion for improvement was the key differentiator between high- and moderate-quality feedback. The overall QuAL model predicted the exact human-rated score or 1 point above or below it in 87% of instances. Overall model performance was excellent, especially regarding the subtasks on suggestions for improvement and the link between resident performance and improvement suggestions, which achieved 85% and 82% balanced accuracies, respectively. CONCLUSIONS: This model could save considerable time for programs that want to rate the quality of supervisor comments, with the potential to automatically score a large volume of comments. This model could be used to provide faculty with real-time feedback or as a tool to quantify and track the quality of assessment comments at faculty, rotation, program, or institution levels.

Considering the Secondary Use of Clinical and Educational Data to Facilitate the Development of Artificial Intelligence Models.

ABSTRACT: Medical training programs and health care systems collect ever-increasing amounts of educational and clinical data. These data are collected with the primary purpose of supporting either trainee learning or patient care. Well-established principles guide the secondary use of these data for program evaluation and quality improvement initiatives. More recently, however, these clinical and educational data are also increasingly being used to train artificial intelligence (AI) models. The implications of this relatively unique secondary use of data have not been well explored. These models can support the development of sophisticated AI products that can be commercialized. While these products have the potential to support and improve the educational system, there are challenges related to validity, patient and learner consent, and biased or discriminatory outputs. The authors consider the implications of developing AI models and products using educational and clinical data from learners, discuss the uses of these products within medical education, and outline considerations that should guide the appropriate use of data for this purpose. These issues are further explored by examining how they have been navigated in an educational collaborative.

Deidentifying Narrative Assessments to Facilitate Data Sharing in Medical Education.

PROBLEM: Narrative assessments are commonly incorporated into competency-based medical education programs. However, efforts to share competency-based medical education assessment data among programs to support the evaluation and improvement of assessment systems have been limited in part because of security concerns. Deidentifying assessment data mitigates these concerns, but deidentifying narrative assessments is time-consuming, resource intensive, and error prone. The authors developed and tested a tool to automate the deidentification of narrative assessments and facilitate their review. APPROACH: The authors met throughout 2021 and 2022 to iteratively design, test, and refine the deidentification algorithm and data review interface. Preliminary testing of the prototype deidentification algorithm was performed using narrative assessments from the University of Saskatchewan emergency medicine program. The algorithm's accuracy was assessed by the authors using the review interface designed for this purpose. Formal testing included 2 rounds of deidentification and review by members of the authorship team. Both the algorithm and data review interface were refined during the testing process. OUTCOMES: Authors from 3 institutions, including 3 emergency medicine programs, an anesthesia program, and a surgical program, participated in formal testing. In the final round of review, 99.4% of the narrative assessments were fully deidentified (names, nicknames, and pronouns removed). The results were comparable for each institution and specialty. The data review interface was improved with feedback obtained after each round of review and found to be intuitive. NEXT STEPS: This innovation has demonstrated viability evidence of an algorithmic approach to the deidentification of assessment narratives while reinforcing that a small number of errors are likely to persist. Future steps include the refinement of both the algorithm to improve its accuracy and the data review interface to support additional data set formats.

Convergent adaptation of true crabs (Decapoda: Brachyura) to a gradient of terrestrial environments.

For much of terrestrial biodiversity, the evolutionary pathways of adaptation from marine ancestors are poorly understood, and have usually been viewed as a binary trait. True crabs, the decapod crustacean infraorder Brachyura, comprise over 7,600 species representing a striking diversity of morphology and ecology, including repeated adaptation to non-marine habitats. Here, we reconstruct the evolutionary history of Brachyura using new and published sequences of 10 genes for 344 tips spanning 88 of 109 brachyuran families. Using 36 newly vetted fossil calibrations, we infer that brachyurans most likely diverged in the Triassic, with family-level splits in the late Cretaceous and early Paleogene. By contrast, the root age is underestimated with automated sampling of 328 fossil occurrences explicitly incorporated into the tree prior, suggesting such models are a poor fit under heterogeneous fossil preservation. We apply recently defined trait-by-environment associations to classify a gradient of transitions from marine to terrestrial lifestyles. We estimate that crabs left the marine environment at least seven and up to 17 times convergently, and returned to the sea from non-marine environments at least twice. Although the most highly terrestrial- and many freshwater-adapted crabs are concentrated in Thoracotremata, Bayesian threshold models of ancestral state reconstruction fail to identify shifts to higher terrestrial grades due to the degree of underlying change required. Lineages throughout our tree inhabit intertidal and marginal marine environments, corroborating the inference that the early stages of terrestrial adaptation have a lower threshold to evolve. Our framework and extensive new fossil and natural history datasets will enable future comparisons of non-marine adaptation at the morphological and molecular level. Crabs provide an important window into the early processes of adaptation to novel environments, and different degrees of evolutionary constraint that might help predict these pathways.

The rebel, the professor, and the entrepreneur: Qualitative study to explore creator stories of FOAM.

Introduction: The free open access medical education (#FOAMed, or FOAM) movement creates educational content intended to inform medical education and clinical practice and is distributed in an unrestricted fashion (e.g., open access website). The who, what, and in particular the how of FOAM has raised important questions about the sustainability of the movement. Methods: We recruited a diverse research team that included educational researchers, FOAM contributors, a business academician, and medical trainees to design and conduct a qualitative study exploring the work of FOAM creators. We analyzed the transcripts of interviews with 11 participants from top FOAM websites in emergency medicine and critical care. The team met frequently to iteratively identify and discuss emergent themes (major and minor) until saturation of concepts was achieved. Results: Creators of FOAM could be categorized using three archetypes: the rebel, the professor, and the entrepreneur. The rebel was categorized as distinctly rejecting "traditional academic structures" yet was compelled to deliver educational content via alternative routes. The professor retained a traditional academic role, instead creating FOAM to supplement academic activities (teaching courses, disseminating scholarship, promotion). Entrepreneurs focused on creating a sustainable entity in an effort to supplement their income and reduce clinical obligations. Conclusion: While all FOAM creators appear unified in their passion to create, promote, and distribute educational material with unfettered access to educators, their motivations for creating content could be differentiated. Given the grassroots nature of FOAM, creators share concerns related to financing, time commitments, and threats to sustainability of these businesses. The longevity of FOAM and what business models are best suited to support them are uncertain. Further exploration of the implications could investigate the best ways to engage with and support the different FOAM creator archetypes and develop models of sustainability.

Call to action: equity, diversity, and inclusion in emergency medicine resident physician selection.

OBJECTIVES: This call to action seeks to improve emergency care in Canada for equity-deserving communities, enabled by equitable representation among emergency physicians nationally. Specifically, this work describes current resident selection processes and makes recommendations to enhance the equity, diversity, and inclusion (EDI) of resident physician selection in Canadian emergency medicine (EM) residency programs. METHODS: A diverse panel of EM residency program directors, attending and resident physicians, medical students, and community representatives met monthly from September 2021 to May 2022 via videoconference to coordinate a scoping literature review, two surveys, and structured interviews. This work informed the development of recommendations for incorporating EDI into Canadian EM resident physician selection. At the 2022 Canadian Association of Emergency Physicians (CAEP) Academic Symposium, these recommendations were presented to symposium attendees composed of national EM community leaders, members, and learners. Attendees were divided into small working groups to discuss the recommendations and address three conversation-facilitating questions. RESULTS: Symposium feedback informed a final set of eight recommendations to promote EDI practices during the resident selection process that address recruitment, retention, mitigating inequities and biases, and education. Each recommendation is accompanied by specific, actionable sub-items to guide programs toward a more equitable selection process. The small working groups also described perceived barriers to the implementation of these recommendations and outlined strategies for success that are incorporated into the recommendations. CONCLUSION: We call on Canadian EM training programs to implement these eight recommendations to strengthen EDI practices in EM resident physician selection and, in doing so, help to improve the care that patients from equity-deserving groups receive in Canada's emergency departments (EDs).

ABSTRAIT: OBJECTIFS: Cet appel à l'action vise à améliorer les soins d'urgence au Canada pour les collectivités méritant l'équité, grâce à une représentation équitable parmi les médecins d'urgence à l'échelle nationale. Plus précisément, ce travail décrit les processus actuels de sélection des médecins résidents et formule des recommandations pour améliorer l'équité, la diversité et l'inclusion (EDI) de la sélection des médecins résidents dans les programmes de résidence en médecine d'urgence (SE) du Canada. MéTHODES: Un groupe diversifié de directeurs du programme de résidence en GU, de médecins résidents, d'étudiants en médecine et de représentants communautaires se sont réunis mensuellement de septembre 2021 à mai 2022 par vidéoconférence pour coordonner une analyse documentaire, deux sondages et des entrevues structurées. Ces travaux ont orienté l'élaboration de recommandations pour l'intégration de l'IDE dans la sélection des médecins résidents en SE au Canada. À l'occasion du Symposium universitaire 2022 de l'Association canadienne des médecins d'urgence (ACMU), ces recommandations ont été présentées aux participants au symposium composé de dirigeants, de membres et d'apprenants de la communauté nationale de la GU. Les participants ont été divisés en petits groupes de travail pour discuter des recommandations et aborder trois questions facilitant la conversation. RéSULTATS: Les commentaires recueillis lors du symposium ont servi à formuler une dernière série de huit recommandations visant à promouvoir les pratiques de l'IDE au cours du processus de sélection des résidents qui traitent du recrutement, du maintien en poste, de l'atténuation des inégalités et des préjugés, et de l'éducation. Chaque recommandation est accompagnée de sous-éléments précis et réalisables pour orienter les programmes vers un processus de sélection plus équitable. Les petits groupes de travail ont également décrit les obstacles perçus à la mise en œuvre de ces recommandations et décrit les stratégies de réussite qui sont intégrées aux recommandations. CONCLUSION: Nous demandons aux programmes canadiens de formation en GU de mettre en œuvre ces huit recommandations afin de renforcer les pratiques d'IDE dans la sélection des médecins résidents en GU et, ce faisant, d'aider à améliorer les soins que les patients des groupes méritant l'équité reçoivent dans les services d'urgence du Canada.

Digital Impact Factor: A Quality Index for Educational Blogs and Podcasts in Emergency Medicine and Critical Care.

STUDY OBJECTIVE: Given the popularity of educational blogs and podcasts in medicine, learners and educators need tools to identify trusted and impactful sites. The Social Media Index was a multi-sourced formula to rank the effect of emergency medicine and critical care blogs. In 2022, a key data point for the Social Media Index became unavailable. This bibliometric study aimed to develop a new measure, the Digital Impact Factor, as a replacement. METHODS: The Digital Impact Factor incorporated modern measures of website authority and reach. This formula was applied to a cross-sectional study of active emergency medicine and critical care blogs and podcasts. For each website, we generated a Digital Impact Factor score based on Ahrefs Domain Rating and the follower count of the websites' pages from 8 social media platforms. A series of Spearman correlations provided evidence of association by comparing a rank-ordered list to rank lists derived from the Social Media Index over the last 5 years. The Bland-Altman analysis assessed for agreement. RESULTS: The authors identified 88 relevant websites with a median Ahrefs Domain Rating of 28 (range 0 to 71, maximum 100) and total social media followership count across 8 platforms of 1,828,557. The Domain Rating and individual social media followership scores were normalized based on the highest recorded values to yield the Digital Impact Factor (median 4.57; range 0.02 to 9.50, maximum 10). The correlation between the 2022 Digital Impact Factor and the 2021 Social Media Index was 0.94 (95% confidence interval 0.89 to 0.97; p<.001; n=41 rankings correlated), suggesting that they measure similar constructs. The Bland-Altman plot also demonstrated fair agreement between the 2 scores. CONCLUSION: The Digital Impact Factor is a measure of the relative effect of educational blogs and podcasts within emergency medicine and critical care.

Guidelines: The Do's, Don'ts and Don't Knows of Creating Open Educational Resources.

Background: In medical education, there is a growing global demand for Open Educational Resources (OERs). However, OER creators are challenged by a lack of uniform standards. In this guideline, the authors curated the literature on how to produce OERs for medical education with practical guidance on the Do's, Don'ts and Don't Knows for OER creation in order to improve the impact and quality of OERs in medical education. Methods: We conducted a rapid literature review by searching OVID MEDLINE, EMBASE, and Cochrane Central database using keywords "open educational resources" and "OER". The search was supplemented by hand searching the identified articles' references. We organized included articles by theme and extracted relevant content. Lastly, we developed recommendations via an iterative process of peer review and discussion: evidence-based best practices were designated Do's and Don'ts while gaps were designated Don't Knows. We used a consensus process to quantify evidentiary strength. Results: The authors performed full text analysis of 81 eligible studies. A total of 15 Do's, Don't, and Don't Knows guidelines were compiled and presented alongside relevant evidence about OERs. Discussion: OERs can add value for medical educators and their learners, both as tools for expanding teaching opportunities and for promoting medical education scholarship. This summary should guide OER creators in producing high-quality resources and pursuing future research where best practices are lacking.